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Application/Control Number: 09/224,009 
Art Unit: 2176 

DETAILED ACTION 

1. This action is responsive to communications: amendment filed 5/6/2003, to the original 
application filed 12/31/1998. 

2. The rejection of claims 1-9, 1 1-32, 34-44 under 35 U.S.C 103(a) as being unpatentable over Balogh has 
been withdrawn as necessitated by amendment. 

5. Claims 1-9, 11-12, 14-32, 34-35, 37-44 are pending. Claims 13, 36 have been canceled by Applicant 
Claims 1, 16, 24, 38 are independent claims. 

Claim Rejections - 35 USC § 103 

7. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness 
rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the 
differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been 
obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall 
not be negatived by the manner in which the invention was made. 

8. Claims 1-9, 11-12, 14-32, 34-35, 37-44 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Balogh et al. (hereinafter Balogh), U.S. Patent No. 5,493,677 issued February 20, 1996, in view of Cox 
et al. (hereinafter Cox), U.S. Patent No. 5,696,964 issued December 09, 1997 (originally referenced in a 
previous action). 

In regard to independent claim 1, Balogh teaches an application program comprising plurality of 
media clips in a database, with associated information describing each media clip, said media clips are in the 
form of images, as well as video clips and multimedia objects (Balogh Abstract, column 1 lines 56-64, column 3 
lines 29-34, column 5 lines 48-57, Figure 6, 14; compare with claim 1 "A method for searching a media clip 
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database associated with a multimedia application program, wherein said media clip database contains,.. . that 
describes each associated media clip in said media clip database, comprising;". 

Balogh teaches a "captioner" which provides metadata in the form of a caption describing salient 
features of an image, bibliographic data, "suggest fields" and attributes of said image, for each image (Balogh 
column 3 lines 20-43). Since said data can include descriptive words (i.e. caption "blue collar" - Balogh column 
6 lines 38-44), and since Balogh teaches that keyword searching can be applied to querying (Balogh column 12 
lines 8-14), as well as teaching that captions/bibliographic information can be reused for iterative querying 
(Balogh column 14 lines 40-60), said teachings provide a reasonable suggestion to one of ordinary skill in the 
art at the time of the invention, of the use of said metadata data as keywords, providing Balogh the benefit of 
querying with descriptive keywords for searching various image databases (compare with claim 1 "keywords 79 ). 

Balogh teaches a user performing an initial query (Balogh column 1 1 lines 60-67, column 12 lines 1-7), 
resulting in retrieval of captions with images along with associated information (Balogh column 14 lines 3-10, 
column 16 lines 42-47, 56-67, Figure 12, 13) (compare with claim 1 "(a) in response to a user selecting a media 
clip, retrieving information... . associated with said selected media clip from said media clip database"). 

Balogh teaches presenting the above captions, images and information to a user for eventual query 
(Balogh column 14 lines 3-10, 40-60, Figure 12-15; compare with claim 1 "(b) simultaneously presenting to the 
user for selection by the user: "(i) said keywords associated with said media clips;"). . 

Balogh does not specifically teach hidden criteria. However, Cox Cox teaches PicHunter, a media 
searching tool comprising a GUI interface with four displayed images (clips) from a clip database. A user 
selects an image, then optionally selects button "GO", after which said invention searches and presents a second 
set of images similar to what was initially selected (Cox Figure 2, column 5 lines 29-45, column 8 lines 19- 
28). Cox's invention involves the analyzation of a user selected image, so that a similar set of images can be 
retrieved/presented In choosing an image, the image attributes (i.e. color, shape, etc.) inherent within said 
image, along with various noted (hidden) features of each image (i.e. contrast, saturation, etc.) , are used as 



Application/Control Number: 09/224,009 Page 4 

Art Unit: 2176 

search criteria by the system, (Cox Abstract, column 5 lines 35-45, 62-67, column 6 lines 1-23; compare with 
claim 1 "and hidden criteria associated with find similar clips indicia 7 ', "(ii) said find similar clips indicia 
having associated hidden criteria", and "hidden criteria"). It would have been obvious to one of ordinary skill 
in the art at the time of the invention to apply Cox to Balogh, providing Balogh the benefit of intelligent 
(accurate) analyzation of media images incorporating hidden search criteria, which may not be explicitly present 
in a user's vocabulary (or known to the user) (see Cox column 1 lines 45-47). 

Balogh teaches an additional query based upon the associated image description or bibliographic data of 
a previously selected image, the user can make further additions, deletions, and/or modifications to the 
associated information, if needed, prior to said additional query. It is to be noted that a user drags and drops a 
selected image into the description/bibliographic area, resulting in transfer of associated information (i.e. 
keywords, as explained above) to be used or edited in the additional query, resulting in retrieval of additional 
images (Balogh column 14 lines 40-59, column 16 lines 53-62; compare with claim 1 "(c) in response to the 
user selecting a search criteria by selecting one or more of said keywords and/or said find similar clips indicia 
associated with said selected media clip, retrieving all media clips in said media clip database that match the 
search criteria created by the user"). 

In regard to dependent claims 2-4, Balogh teaches display of the best retrieved captions/ images 
displayed for user review. This process (including displaying a plurality of clip images) is repeated by said user, 
as desired (Balogh Figure 13-15, column 14 lines 3-10, 40-42, column 16 lines 53-62; compare with claims 2- 
4). 

In regard to dependent claims 5-8, Balogh teaches a media type (images), as well as search criteria 
based upon color and/or shape (i.e. "red trucks", and "black cats" - keywords) (Balogh Abstract, Figure 2, 
column 1 1 lines 60-67; compare with claim 5). Balogh does not sepecifically teach hidden criteria as artistic 
style, shape, and color. However, Cox teaches the analyzation of a user selected image, so that a similar set of 
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images can be retrieved/presented In choosing an image, the image attributes (i.e. color, shape, etc.) inherent 
within said image, along with various noted (hidden) features of each image (i.e. pixel color, saturation, contrast, 
and image width (shape), etc. - suggestive of artistic styles), are used as search criteria by the system, (Cox 
Abstract, column 5 lines 35-45, 62-67, column 6 lines 1-23; compare with claims 6-8. It would have been 
obvious to one of ordinary skill in the art at the time of the invention to apply Cox to Balogh, providing Balogh 
the benefit of intelligent (accurate) analyzation of media images incorporating hidden search criteria, which may 
not be explicitly present in a user's vocabulary (or known to the user) (see Cox column 1 lines 45-47). 

In regard to dependent claim 9, claim 9 reflects the combined subject matter of claims 7 and 8, and is 
rejected along the same rationale. 

In regard to dependent claims 11-12, Balogh teaches an additional query, whereby a result image is 
dragged and dropped into a description/bibliographic area, resulting in bibliographic (keyword) data copied to 
form a new query (in the case of Balogh, all of the data is selected) (Balogh column 14 lines 40-59; compare 
with claim 11). 

Balogh teaches search criteria in the form of keywords as part of a search query, said query (keywords) 
can be based upon image characteristics or image type (Balogh column 1 1 lines 60-67, column 12 lines 9-14; 
compare with claim 12). 

In regard to dependent claim 14, Balogh teaches dragging and dropping of a selected image into a 
description/bibliographic area, resulting in a copy of the image's bibliographic data (keywords) into formulation 
of a new query. Balogh also teaches searching and presentation of files of type image from an image database 
(Balogh Abstract, column 14 lines 49-59; compare with claim 14). 
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In regard to dependent claim 15, a computer readable medium (i.e. diskette or hard drive) used for 
holding instructions is known in the art. 

In regard to independent claim 16, JBalogh teaches an application program comprising plurality of 
media clips in a database, with associated information describing each media clip, said media clips are in the 
form of images, as well as video clips and multimedia objects. It is to be noted that Balogh also discloses 
associated data regarding what a particular media image suggests, which is indicative of a visual thesaurus 
(Balogh Abstract, column 1 lines 56-64, column 3 lines 29-34, column 5 lines 48-57, Figure 6, 14, see also 
Figure 3 item 262, Figure 6 item 606, column 1 lines 59-61, column 3 lines 33-36; compare with claim 16 "A 
method for providing a user interface for a visual thesaurus for a media clip database associated with a 
multimedia application program, wherein said media clip database contains information that describes each 
associated media clip in said media clip database, comprising", 

Balogh teaches a "captioner" which provides metadata in the form of a caption describing salient 
features of an image, bibliographic data, "suggest fields" and attributes of said image, for each image (Balogh 
column 3 lines 20-43). Since said data can include descriptive words (i.e. caption "blue collar" - Balogh column 
6 lines 38-44), and since Balogh teaches that keyword searching can be applied to querying (Balogh column 12 
lines 8-14), as well as teaching that captions/bibliographic information can be reused for iterative querying 
(Balogh column 14 lines 40-60), said teachings provide a reasonable suggestion to one of ordinary skill in the 
art at the time of the invention, of the use of said metadata data as keywords, providing Balogh the benefit of 
querying with descriptive keywords for searching various image databases (compare with claim 16 "keywords"). 

Balogh teaches a user performing an initial query (Balogh column 1 1 lines 60-67, column 12 lines 1-7), 
resulting in retrieval of captions with images along with associated information presenting said captions, images 
and information to a user (Balogh column 14 lines 3-10, 40-41, column 16 lines 42-47, 56-67, Figure 12-15) 
(compare with claim 16 "directly in response to a user selecting a media clip from said media clip database"). 
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Balogh teaches an additional query based upon the associated image description or bibliographic data of 
a previously selected image, the user can make further additions, deletions, and/or modifications to the 
associated information, if needed, prior to said additional query. It is to be noted that a user has the option of 
visually dragging and dropping a selected image into the desaiption/bibliographic area, resulting in transfer of 
associated information to be used or edited in the additional query, resulting in retrieval of additional images, 
said drag and drop is a visual indication of said option (Balogh column 14 lines 40-59, column 16 lines 53-62; 
compare with claim 16 "displaying to the user an option for finding similar media clips that have an associated 
keyword that matches the associated keyword for the selected clip."). 

Balogh does not specifically teach hidden criteria. However, Cox Cox teaches PicHunter, a media 
searching tool comprising a GUI interface with four displayed images (clips) from a clip database. A user 
selects an image, then optionally selects button "GO", after which said invention searches and presents a second 
set of images similar to what was initially selected (Cox Figure 2, column 5 lines 29-45, column 8 lines 19- 
28). Cox's invention involves the analyzation of a user selected image, so that a similar set of images can be 
retrieved/presented. In choosing an image, the image attributes (i.e. color, shape, etc.) inherent within said 
image, along with various noted (hidden) features of each image (i.e. contrast, saturation, etc.) , are used as 
search criteria by the system, (Cox Abstract, column 5 lines 35-45, 62-67, column 6 lines 1-23; compare with 
claim 16 "hidden criteria associated with find similar clips indicia", "find similar clips indicia hidden criteria"). 
It would have been obvious to one of ordinary skill in the art at the time of the invention to apply Cox to Balogh, 
providing Balogh the benefit of intelligent (accurate) analyzation of media images incorporating hidden search 
criteria, which may not be explicitly present in a user's vocabulary (or known to the user) (see Cox column 1 
lines 45-47). 
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In regard to dependent claim 17, Balogh teaches a browsing tool for allowing a user to visually 
browse hits, as well as a number of "select" buttons for choosing certain candidate matches for further 
examination (Balogh column 16 lines 42-47, 55-60). 

In regard to dependent claim 18, Balogh teaches an additional query based upon the associated image 
description or bibliographic data of a previously selected image, the user can make further additions, deletions, 
and/or modifications to the associated information, if needed, prior to said additional query. It is to be noted that 
a user has the option of visually dragging and dropping a selected image into the description/bibliographic area, 
resulting in transfer of associated information to be used or edited in the additional query, resulting in retrieval 
of additional images, said drag and drop is a visual indication of said option (Balogh column 14 lines 40-59, 
column 16 lines 53-62). 

In regard to dependent claims 19-21, 23, a fly-out window (i.e. an overlaying window, or balloon 
help annotation with additional information, etc.), is known in the art (compare with claim 19). 

An option for inserting an image into a document (i.e. clipboard copy and paste), is known in the 
document processing art (compare with claim 20). 

Balogh teaches a browser for vewing image hits, said hits comprise a thumbnail (preview) image along 
with a caption "snippet" from each image (Balogh column 16 lines 63-67, column 17 lines 1-13; compare with 
claim 21). 

A computer readable medium (i.e. diskette or hard drive) used for holding instructions is known in the 
art (compare with claim 23). 



In regard to dependent claim 22, Balogh does not specifically teach an option to add a clip to a 
category. However, this limitation would have been obvious to one of ordinary skill in the art at the time of the 
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invention, in view of Balogh, because Balogh teaches ordering selected images through a purchase/delivery 
service (Balogh column 17 lines 40-44). In the case of Balogh, images are user selected for purchase (a user 
voluntarily adding a media clip to a category intended for purchase) (see also Balogh column 17 lines 48-50, 
and column 18 lines 1-10). Adapting Balogh to incorporate user inclusion of clips into various additional 
categories as taught by Balogh, provides a user of Balogh the benefit of grouping selected images for further 
action (i.e. negotiation, reservation, trade, etc.). 

In regard to independent claim 24, Balogh teaches a data entry, disambiguation, and database 
processors within a SUN SPARCSTATION (Balogh column 3 lines 60-67; compare with claim 24 "a 
processing unit"). 

Balogh teaches a computer readable medium (i.e. diskette or hard drive) used for holding instructions 
and utilized within a computer, is known in the art (compare with claim 24 "a storage medium... .by the 
processing for"). 

Balogh teaches an application program comprising plurality of media clips in a database, with 
associated information describing each media clip, said media clips are in the form of images, as well as video 
clips and multimedia objects, said clips subject to user query and selection (Balogh Abstract, column 1 lines 56- 
64, column 3 lines 29-34, column 5 lines 48-57, column 16 lines 56-67, column 17 lines 1-13, Figure 6, 14; 
compare with claim 24 ''providing an interface for a user to select a media clip... .in said media clip database"). 

Balogh teaches a "captioner" which provides metadata in the form of a caption describing salient 
features of an image, bibliographic data, "suggest fields" and attributes of said image, for each image (Balogh 
column 3 lines 20-43). Since said data can include descriptive words (i.e. caption "blue collar" - Balogh column 
6 lines 38-44), and since Balogh teaches that keyword searching can be applied to querying (Balogh column 12 
lines 8-14), as well as teaching that captions/bibliographic information can be reused for iterative querying 
(Balogh column 14 lines 40-60), said teachings provide a reasonable suggestion to one of ordinary skill in the 
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art at the time of the invention, of the use of said metadata data as keywords, providing Balogh the benefit of 
querying with descriptive keywords for searching various image databases (compare with claim 24 "keywords"). 

Balogh does not specifically teach hidden criteria. However, Cox Cox teaches PicHunter, a media 
searching tool comprising a GUI interface with four displayed images (clips) from a clip database. A user 
selects an image, then optionally selects button "GO", after which said invention searches and presents a second 
set of images similar to what was initially selected. (Cox Figure 2, column 5 lines 29-45, column 8 lines 19- 
28). Cox's invention involves the analyzation of a user selected image, so that a similar set of images can be 
retrieved/presented In choosing an image, the image attributes (i.e. color, shape, etc.) inherent within said 
image, along with various noted (hidden) features of each image (i.e. contrast, saturation, etc.) , are used as 
search criteria by the system, (Cox Abstract, column 5 lines 35-45, 62-67, column 6 lines 1-23; compare with 
claim 24 "(i) ... including hidden criteria associated with find similar clips indicia", "(ii) find similar clips 
indicia hidden criteria", and "(Hi) find similar clips indicia hidden criteria" ). It would have been obvious to 
one of ordinary skill in the art at the time of the invention to apply Cox to Balogh, providing Balogh the benefit 
of intelligent (accurate) analyzation of media images incorporating hidden search criteria, which may not be 
explicitly present in a user's vocabulary (or known to the user) (see Cox column 1 lines 45-47). 

Balogh teaches a user performing an initial query (Balogh column 1 1 lines 60-67, column 12 lines 1-7), 
resulting in retrieval of captions with images along with associated information, as well as an additional query 
based upon the associated image description or bibliographic data of a previously selected image, the user can 
make further additions, deletions, and/or modifications to the associated information, if needed, prior to said 
additional query. It is to be noted that a user drags and drops a selected image into the description/bibliographic 
area, resulting in transfer of associated information to be used or edited in the additional query, resulting in 
retrieval of additional images (Balogh column 14 lines 40-59, column 16 lines 42-47, 53-67, Figures 12-13; 
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compare with claim 24 providing an interface for the user to select... .with said selected media clip", and "in 
response to the user... for the selected media clip."). 

In regard to dependent claims 25-32, 34-35, 37, claims 25-32, 34-37 reflect the apparatus comprising 
computer readable instructions used for performing the methods as claimed in claims 2-9, 11-12, 14, 
respectively, and are rejected along the same rationale. 

In regard to independent claim 38, Balogh teaches an application program comprising plurality of 
media clips in a database, with associated information describing each media clip, said media clips are in the 
form of images, as well as video clips and multimedia objects. It is to be noted that Balogh also discloses 
associated data regarding what a particular media image suggests, which is indicative of a visual thesaurus 
(Balogh Abstract, column 1 lines 56-64, column 3 lines 29-34, column 5 lines 48-57, Figure 6, 14, see also 
Figure 3 item 262, Figure 6 item 606, column 1 lines 59-61, column 3 lines 33-36; compare with claim 38 "An 
apparatus for providing a user interface for a visual thesaurus for a media clip database associated with a 
multimedia application program, wherein said media clip database contains information... . that describes each 
associated media clip in said media clip database, comprising", 

Balogh teaches a "captioner" which provides metadata in the form of a caption describing salient 
features of an image, bibliographic data, "suggest fields" and attributes of said image, for each image (Balogh 
column 3 lines 20-43). Since said data can include descriptive words (i.e. caption "blue collar" - Balogh column 
6 lines 38-44), and since Balogh teaches that keyword searching can be applied to querying (Balogh column 12 
lines 8-14), as well as teaching that captior^ibliographic information can be reused for iterative querying 
(Balogh column 14 lines 40-60), said teachings provide a reasonable suggestion to one of ordinary skill in the 
art at the time of the invention, of the use of said metadata data as keywords, providing Balogh the benefit of 
querying with descriptive keywords for searching various image databases (compare with claim 38 "keywords"). 
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Balogh teaches a data entry, disambiguation, and database processors within a SUN SPARCSTATION 
(Balogh column 3 lines 60-67; compare with claim 38 "a processing unit"). 

Balogh teaches a computer readable medium (i.e. diskette or hard drive) used for holding instructions 
and utilized within a computer, is known in the art (compare with claim 38 "a storage medium.... by the 
processing unit for,."). 

Balogh teaches a user performing an initial query (Balogh column 1 1 lines 60-67, column 12 lines 1-7), 
resulting in retrieval of captions with images along with associated information presenting said captions, images 
and information to a user, as well as an additional query based upon the associated image description or 
bibliographic data of a previously selected image, the user can make further additions, deletions, and/or 
modifications to the associated information, if needed, prior to said additional query. It is to be noted that a user 
has the option of visually dragging and dropping a selected image into the description/bibliographic area, 
resulting in transfer of associated information to be used or edited in the additional query, resulting in retrieval 
of additional images, said drag and drop is a visual indication of said option (Balogh column 14 lines 3-10, 40- 
59, column 16 lines 42-47, 53-67; compare with claim 38 "... displaying to the user an option for finding 
similar media clips that have associated keywords that matches the associated keywords for a selected media 
clip, directly in response to the user selecting the media clip"). 

Balogh does not specifically teach hidden criteria. However, Cox Cox teaches PicHunter, a media 
searching tool comprising a GUI interface with four displayed images (clips) from a clip database. A user 
selects an image, then optionally selects button "GO", after which said invention searches and presents a second 
set of images similar to what was initially selected (Cox Figure 2, column 5 lines 29-45, column 8 lines 19- 
28). Cox's invention involves the analyzation of a user selected image, so that a similar set of images can be 
retrieved/presented In choosing an image, the image attributes (i.e. color, shape, etc.) inherent within said 
image, along with various noted (hidden) features of each image (i.e. contrast, saturation, etc.) , are used as 
search criteria by the system, (Cox Abstract, column 5 lines 35-45, 62-67, column 6 lines 1-23; compare with 
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claim 38 "hidden criteria associated with find similar clips indicia 77 , and "find similar clips indicia hidden 
criteria"). It would have been obvious to one of ordinary skill in the art at the time of the invention to apply Cox 
to Balogh, providing Balogh the benefit of intelligent (accurate) analyzation of media images incorporating 
hidden search criteria, which may not be explicitly present in a user's vocabulary (or known to the user) (see 
Cox column 1 lines 45-47). 

In regard to dependent claims 39-43, claims 39-43 reflect the apparatus comprising computer readable 
instructions used for performing the methods as claimed in claims 17-21, respectively, and are rejected along the 
same rationale. 

In regard to dependent claim 44, claim 44 reflects the apparatus comprising computer readable 
instructions used for performing the methods as claimed in claim 22, and is rejected along the same rationale. 



Response to Arguments 

10. Applicants arguments with respect to the instant claims have been carefully and fully considered but are 
not persuasive. 

Applicant's arguments regarding newly amended subject matter (i.e. "hidden characteristics 77 , etc.) are 
currently moot in view of the new grounds of rejections, reintroducing the Cox reference. 

Applicant's arguements on pages 10-15 of the amendment are substantially directed to the assertion that 
Balogh fails to teach keywords. The examiner respectfully notes that Balogh's "captioner" provides each image 
with metadata in the form of a caption describing salient features of an image, bibliographic data, "suggest 
fields" and attributes, for each image, since said metadata can include descriptive words, as well as teaching that 
captions/bibliographic information can be reused for iterative querying (see Balogh Figure 2). Since Balogh 



Application/Control Number: 09/224,009 Page 14 

Art Unit: 2176 

teaches that keyword searching can be applied to querying (Balogh column 12 lines 8-14) said teachings provide 
a reasonable suggestion to one of ordinary skill in the art at the time of the invention, of the use of said metadata 
data as keywords. In addition, the metadata is displayed to a user for subsequent querying of images. Although 
Balogh teaches an embodiment comprising natural language sentence captions, nevertheless, the words of said 
sentences are suggestive of keywords, since Balogh teaches parsing a query into individual tokens representing 
single words or multi- words, said words subsequently matched to various captions (as well as other metadata) 
within the search process (see Balogh column 12 lines 33-37). 



Conclusion 

1 1. Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. 
Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the 
extension of time policy as set forth in 37 CFR 1. 136(a). 

A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the 
mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this 
final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory 
period, then the shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1. 136(a) will be calculated from the mailing date of the advisory action. In no 
event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final 
action. 
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12. Any inquiry concerning this communication or earlier communications from the examiner 
should be directed to William Bashore whose telephone number is (703) 308-5807. The examiner can 
normally be reached on Monday through Friday from 1 1 :30 AM to 8:00 PM EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, 
Joseph Feild, can be reached on (703) 305-9792. 

Any inquiry of a general nature or relating to the status of this application should be directed to 
the Group receptionist whose telephone number is (703) 305-3900. 

13. Any response to this action should be mailed to: 



Commissioner of Patents and Trademarks 
Washington, D.C. 2023 1 



or faxed to: 



(703) 746-7239 (for formal communications intended for entry) 



or: 



(703) 746-7240 (for informal or draft communications, please label 
"PROPOSED" or "DRAFT") 



or: 



(703) 746-7238 (for after-final communications) 



Hand-delivered responses should be brought to Crystal Park II, 2121 Crystal Drive, 
Arlington, VA, Fourth Floor (Receptionist). 



William L. Bashore 
•July 10, 2003 




SAMJfV SHAH 



